1. Technical Field
The present application relates generally to an improved data processing system and method. More specifically, the present application is directed to a system and method for balancing PCI-Express Bandwidth across a plurality of PCI-Express adapters.
2. Description of Related Art
Most modern computing devices make use of input/output (I/O) adapters and buses that utilize some version or implementation of the Peripheral Component Interconnect standard, which was originally created by Intel in the 1990s. The Peripheral Component Interconnect (PCI) standard specifies a computer bus for attaching peripheral devices to a computer motherboard. PCI-Express, or PCIe, is an implementation of the PCI computer bus that uses existing PCI programming concepts, but bases the computer bus on a completely different and much faster serial physical-layer communications protocol. The physical layer consists, not of a bi-directional bus which can be shared among a plurality of devices, but of single uni-directional links which are connected to exactly two devices.
FIG. 1 is an exemplary diagram illustrating a PCI-Express (PCIe) fabric topology in accordance with the PCIe specification. As shown in FIG. 1, the PCIe fabric topology 100 is comprised of a host processor (CPU) complex 110 and memory 120 coupled to a root complex or multi-root complex 130, i.e. an I/O North Bridge, which is in turn coupled to one or more of a PCIe endpoint 140 (the term “endpoint” is used in the PCIe specification to refer to PCIe enabled I/O adapters), a PCI express to PCI bridge 150, and one or more interconnect switches 160. The root complex 130 denotes the root of an I/O hierarchy that connects the CPU/memory to the I/O adapters. The root complex 130 includes a host bridge, zero or more root complex integrated endpoints, zero or more root complex event collectors, and one or more root ports. Each root port supports a separate I/O hierarchy. The I/O hierarchies may be comprised of a root complex 130, zero or more interconnect switches 160 and/or bridges 150 (which comprise a switch or PCIe fabric), and one or more endpoints, such as endpoints 170 and 182-188. The endpoints 140, 170, and 182-188 may be, for example, Ethernet, SCSI, SAS, or Fibre Channel I/O adapters. For more information regarding PCI and PCIe, reference is made to the PCI and PCIe specifications available from the peripheral component interconnect special interest group (PCI-SiG) website at www.pcisig.com.
The CPU complex 110 comprises one or more processors and memories (not shown) and resides in a Central Electronics Complex (CEC) 190 above the root complex 130. To connect the CEC 190 to the root complex(es) 130, buses 192 are used that are sometimes referred to as font-side buses, e.g., in International Business Machines Corporation (IBM) enterprise servers, these buses are referred to as the GX+buses. The front-side bus 192 has a limited bandwidth that is a known quantity. For example, a front-side bus 192 may be able to sustain 4 Gbytes/sec of bandwidth.
In the currently known configurations, it is possible to string several drawers with endpoints 140, 170, and 182-188 whose total bandwidth can be over the 4 Gbytes/sec bandwidth available on the front-side bus 192 if their PCIe lanes are used to their maximum levels. A “lane” in the PCI standard is a set of differential signal pairs, one pair for transmission and one pair for reception. A “by-N” link in the PCI standard is composed of N lanes, e.g., an “×8” link or slot supports 8 lanes of traffic to/from an I/O adapter. If the traffic via the PCIe lanes of the endpoints exceeds the available bandwidth of the front-side bus 192, it cannot be guaranteed that the desired performance for each I/O adapter will be achieved and issues may arise with isochronous I/O adapters, i.e. I/O adapters whose data transmissions have data packets that are transmitted at an equal time difference between data packet transmissions, e.g., data transmissions associated with time-dependent data, such as real-time voice and video.
PCI-Express attempted to resolve this issue by creating traffic classes and virtual lanes where one can define which traffic has priority and how much buffer space each lane can use. The problem with this solution is that the industry has not attempted to take advantage of this architectural feature. The industry has not attempted to take advantage of multiple traffic classes and virtual lanes because the transition from PCI-X to PCI-Express was to be performed in as transparent a manner as possible with regard to the operating system and firmware. In order to fully take advantage of traffic classes and virtual lanes, the operating system and firmware would have to be aware of these features and new code would need to be written to take advantage of these features. However, rather than making such modifications, Basic Integrated Operating Systems (BIOS) and known operating systems, such as Microsoft Windows and Linux, have not made the necessary modifications to support the multiple traffic classes and virtual lanes.
Moreover, from a hardware standpoint, adding support for multiple traffic classes and multiple virtual lanes requires added complexity to the chip areas such as arbitration and flow control. Furthermore, adding such support requires additional chip real estate since each virtual lane would require its own private storage for sending and receiving data. As a result, hardware manufacturers have not implemented support for multiple virtual lanes and multiple traffic classes.
Rather than taking advantage of multiple traffic classes and virtual lanes, existing PCI-Express adapters only support one virtual channel and one traffic class. Moreover, the primary operating systems, i.e. Microsoft Windows and Linux, only use one traffic channel and one virtual lane. As a result, currently known environments often encounter problems where the total possible bandwidth of a set of PCI-Express adapter cards within an I/O drawer attached to a CPU complex via an I/O hub or switch may far exceed the capabilities of the I/O hub, switch, and/or the front-side bus.